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Argus-II is one of the most successful epiretinal implantation for providing 
visual acuity those who lost their vision sight due to retinitis pigmentosa (RP) 
problem. However, this model faces color recognition issue is observed from 
implanted patients. Hence, it arises whenever electrode fail to retain same 
electrical stimuli property during sensitivity color transition state is occurred 
(especially, blue and purple colors). To resolve this problem, a proper 
handling of electrical stimuli parameters (amplitude, frequency and pulse 
width) is required during patient under every visual impact is possible. 
Addition to this, the individual patient color sensation is recorded in the 
observation state and creates Argus-II dataset to train the machine learning 
algorithm for maintaining phosphene brightness through controlled 
generation of the electrical stimuli. Therefore, in this paper, an automatic 
recognition of color sensation with controlled phosphene brightness using pre- 
trained convolutional neural network (CNNs) framework is proposed. The 
frequency modulated electrical stimulation of retina is purely influence by 


trained CNNs for adjusting amplitude that can retain maximum brightness 
along with clarity in the color sensation. The experimental results shows that 
the proposed system is achieved reasonable improvement in the transition 
color sensation as well as controlled brightness when compared with other 
existing systems. 
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1. INTRODUCTION 

Human eye is always has a significant quality of sensing multi-colors on the image undergoes several 
levels of visual processing [1]. Color aids in the segmentation of the visual scene and increases the saliency of 
visual inputs at lower and higher levels, items viewed in their characteristic colors are linked to color 
knowledge stored in memory [2]—[4]. Color vision is initiated from cone photoreceptors (S, M, and L) which 
are highly sensitive to light wavelength including all ranges (short, medium, and long) respectively. It is 
commonly accepted that color vision encoding is translated from these three types of cones to three separate 
opponent systems early in the visual pathway: red (R), green (G), blue (B), yellow (Y), and black-white [5], 
[6]. Around 30 to 50 million people suffered by the retinal degeneration which is a major root cause of blindness 
globally [7]. Normally, outer layer of the photoreceptors are completely fade-out at the end stage. As results, 
patients deprive their capacity to perceive light and color [8], [9]. Surgical retinal implantation can help restore 
vision by stimulating the remaining inner retinal neurons. However, it follows under lower spatial resolution 
of 16 to 1600 electrodes [10]-[13]. As a result, any method of increasing visual information for object 
recognition and orientation in a given context is beneficial [14]. However, such an approach for providing 
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limited color feeling as an additional dimension of vision in retinal prosthetics. Argus-II is one of the most 
successful epiretinal implantations for restoring eyesight to people who have lost their vision due to retinitis 
pigmentosa (RP). However, implanted individuals have reported that this model has a color recognition 
problem [15]. As a result, it occurs when an electrode fails to maintain the same electrical stimulation property 
during the sensitivity color transition stage [16]-[18]. To tackle this difficulty, adequate handling of electrical 
stimulation characteristics (amplitude, frequency, and pulse width) is essential during the patient's treatment 
under all conceivable visual impacts. Furthermore, during the observation stage, each patient's color experience 
is recorded, resulting in the creation of the Argus-II dataset, which is used to train the machine learning 
algorithm for sustaining phosphene brightness through controlled generation of electrical stimuli. As a result, 
utilizing a pre-trained convolutional neural network (CNNs) framework, an automatic recognition of color 
feeling with controlled phosphene brightness is proposed in this research. The amplitude of frequency 
modulated electrical stimulation of the retina is only influenced by trained CNNs for retaining maximum 
brightness and clarity in the color sense. When compared to other current systems, the experimental findings 
reveal that the suggested system achieves a reasonable improvement in transition color sense as well as 
controlled brightness [19]. 

The organization of paper is given as follows: Section 2 describes overview of the methodology and 
elaborates about Argus-II datasets. Section 3, discuss the experimental results includes electrical stimuli 
adjustment with controlled phosphene brightness, Color sensation changes as a function of stimulated 
frequency, maintain Long-term stability towards color sensation and correlation-based feature extraction 
algorithm. Finally, conclusion and future scope is given in section 6. 


2. MATERIALS AND METHOD 
2.1. Argus-II dataset 

In this paper, mainly focusing on enhance the accurate color restoring process is initiated 
automatically by using pre-trained CNNs framework. The Argus-II is well-known surgical implantation is 
developed to activate ganglion cell which is not affected by retinitis pigmentosa (RP) problem. It consists of 
linear array (6x10) of electrodes are arranged to receive electrical stimuli with respect to visual scenes. This 
set up is placed on both eye of the patient who has affected by RP degradation. In [13] described the 
experimental study to understand the visuality functioning of the Argus-II on implanted patient. Nearly, 7 
interested members (5 female and 2 males) are participated in the testing phase in which daily observation 
details are recorded for further analysis is carried in future (if required). Additionally, different time stamp of 
Argus-II surgical implanted patients are subjected in the study (i.e. years between 2015 and 2019) and patients 
are registered that they have clear vision in their childhood days due to age factor vision perception quality is 
affected which leads into loss of sight. The training features are derived from patient recorded which is applied 
to CNNs for obtaining trained datasets for better color sensation with controlled phosphene brightness is 
achieved [20]. That implies, amount of current passing to each electrode is automatically adjusted based on the 
trained dataset receive from pre-trained CNNs framework [21]—[23]. Usually, the feature extraction of current 
level thresholding is obtained based on patient records observed during the Argus-II test phase. That means, 
patients are advice to give response (‘yes’ or ‘no’) according to their perceived phosphene quality and 
respective electrical stimulus is recorded. They are given six different options to register their response (i.e. 
not detected, dim, dim to medium, medium, medium to bright and bright). Internally, several threshold points 
are generated and process can be optimized after desired target current level is reached. The individual 
electrode current level Uejectrode © 168A) is identified for an automatic recognition of color sensation with 
controlled phosphene brightness [24]. The artificial intelligence (AI) algorithm is programmed to initiate 
automatic formation electrodes in groups (either pairs or quads) based on current level threshold. Thereby, if 
letectrode 18 greater than Ipjectrode threshold then, group formation is restricted. Otherwise, it allows for electrode 
group formation(Ietectrode threshold < Ljsctnode): Because of group formation, minimize the chance of tissues 
damage due to lower value selection of the electrode current. The AI algorithm is more effective to adjust 
electrode current Igject+oqemay arias fluctuations and maintains identical electrical stimuli for entire electrode 
group [25]. 


2.2. Color categories model 

The patients are subjected to understand the color transition state by giving suitable example that 
describes color sensation by classifying into two set namely hue (color detection is wrong due to much 
similarity color shade is observed by patient) and saturation (color detection is wrong due to unawareness about 
the color scaling bands is exhibited by patient). According to the patient response, brightness is adjusted 
manually from external system during testing phase. The RYGB color model is adapted for recognizing color 
shades (eg: pink, red, brown and green). All colors can be labeled using different proportions of fundamental 
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hue perceptions in red (R), yellow (Y), green (G), and blue (B) in this model [18]. In general, color shades of 
blue and yellow are normalized to achieve quality color sensation with controlled phosphene brightness. It is 
denoted by vector sets such as S$; = (by, bo,....b,) and Sz = (1, Y2,--..¥m) where, nand mbe the maximum 
color shades present in the blue and yellow. Because, patient may faces difficulty to differentiate blue and 
yellow shade colors respectively. Even, amount of hue and saturation perception are unable to predict the 
accurate color classification. The saturation scaling range lies between minimum and maximum point (i.e.y = 
(0.2 — 0.5)). Especially, royal purple, lavender purple, royal yellow and lemon yellow comes under blue and 
yellow color shade. All other colors are classified correctly by the patient. 


2.3. Blue and Yellow score calculation 

The blue score is not differentiated between blue and purple because the observed purple colors are 
highly blue biased, and the current study is focused on the yellow-blue color pathway. The data were analysed 
using paired sample t tests. It was done in R with a significance level of 0.05. Curve fits were improved and 
weighted with the inverse of variance. The following pseudo code is representing scaling system used in the 
color categories model to calculate a blue and yellow score for each color sensation is given below: 


AI algorithm for blue and yellow scoring 

#1 Initialization: Ieectroae = 168LA, Sy = (by, bo,...- bn )and Sz = (4, V2)--++ Vm) 
#2 Saturation scaling: 0.2 to 0.5 

#3 for i=1 to No of patients 

#4 Identify brightness of individual patient and compute Igyectrode 

#5 If letectrode S 168, then 

#6 Set electrode group formation and maintain identical electrical stimuli 

#7 else Ieiectrode > 168then, 

#8 Set restrict the group formation 

#9 Update optimized Iejectroaefrom individual patient response 

#10 Update feature training dataset 

#11 IZfy < 0.2then, 

#12 Set S,; = Sy = 1; blue (S,)/yellow (S,)sensation, the hue is unsaturated. 

#13 elseif y = 0.2&&0.5then, 

#14 Set S; = Sy, = 2; blue (S,)/yellow (S,)sensation, hue is significantly reported. 
#15 else if y > 0.5then, 

#16 Set S, = S, = 3; blue (S,)/yellow (S,)sensation, strongly reported. 

#17 — else y = Othen, 

#18 Set S, = S, = 0; No perception 


3. MATERIALS AND METHODOLOGY 

In this part, the qualitative analysis of the suggested model is carried out by various stimulation 
frequencies to address an automated adjustment of electrical stimuli for improving brightness, ensuring proper 
colour recognition. Pre-trained CNNs update feature extraction sets to generate quality training samples from 
Argus-II patient database. It has 7 members (5 females and 2 males) who record daily observations for analysis 
(if required). The proposed approach uses pre-trained CNNs to improve automated colour restoration. 
Experimental validation is performed by varying stimulation settings such as frequency from 6 to 120 Hz and 
pulse width from 0.2 to 2 ms. In this research, Argus-II has 29 electrodes implanted and recognises 11 colours: 
grey, white, yellow, dark yellow, purple, gold, green, blue, pink, black, orange, and brown. 3 of 11 hues (grey, 
black, and white) are chromatic, while 8 are not. 


3.1. Electrical stimuli adjustment with controlled phosphene brightness 

Patient data show that inadequate electrode current causes colour sense problems. Monitoring patient 
reaction and adjusting electoral stimuli helps decrease colour sensory problems. Figure 1 shows patients' 
responses to modulate phosphene brightnes observed by prosthetic eyesight. For experimental consideration 
we used Indian street side purple flower plants taken for color sensation analysis whose indivudal resolution is 
clearly mentioned as (a) 273x 376 pixels, (b) 27x36 pixels, (c) 3x6 pixels, (d) 573x 400 pixels, (e) 64x32 pixels 
and (f) 9x3 pixels respectively. Figure 2 depicts the color sensation response obtained from patients with 
controlled phosphene brightness. According to Argus-II patient response data, two types of colour sensory 
abnormalities are identified: yellow to blue and blue to purple. Figure 2(a) and Figure 2(b) show the stimulated 
pulse wave and current amplitude for medium brightness. Pre-trained CNNs improve electrical stimulation by 
changing electrodes depending on feature sets. Figure 2 (c) shows dominant hues and needed electrodes for 
each. Chromatic colours need more than 5 electrodes, whereas other colours require less. Purple and yellow 
are other popular colours. Figure 2 shows blue-yellow and red-green axes (d). Color and size of circles indicate 
perceived colour and fundamental frequency. 
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Figure |. The importance of simulated prosthetic eyesight is demonstrated. R1: Colored and greyscale 
represents Indian street side purple flower plants consider for color sensation analysis (a) 273x 376 pixels, (b) 
27x36 pixels, (c) 3x6 pixels. R2: Colored and greyscale represents Indian street side yellow flower plants 
consider for color sensation analysis (d) 573x 400 pixels, (e) 64x32 pixels, and (f) 9x3 pixels respectively 
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Figure 2. Color sensation response obtained from patients with controlled phosphene brightness. (a) 
waveform indicates stimulated pulse at medium brightness with frequency modulation; (b) current amplitude 
required to keep medium brightness (Im) as frequency increases (Results observed from 29 electrodes and 
recorded patient’s responses that converge exponentially towards target point, where ‘R’ is equal to 0.94); (c) 
colors dominant and number of electrodes generating each color are represented (chromatic colors are 
required more than 5 electrodes for identification, others colors identify by less than 5 electrodes which is not 
shown) and (d) color space generated by blue-yellow and red-green axis, where purple and yellow color 
shades are often undergo color sensation. The circles' color and size represent the perceived color and its 
fundamental frequency, respectively 


3.2. Color sensation changes as a function of stimulated frequency 

Changing the current amplitude may change the colour sense. In this part, patient responds to varied 
frequency (50, 100, 150) and pulse width (0.5, 1, 1.5 and 2 ms). Figure 3 depicts it, when the stimulated 
frequency shifts from low (less than 60 Hz) to high (greater than 60 Hz), implanted individuals see a yellow- 
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to-blue transition. In Figure 3(a), colour perception shifts from yellow to blue when stimulus frequency is 
between a few Hz and 150 Hz. If many colours are identified in one phosphene, they are represented in 
concentric circles, each signifying hue and saturation. Figure 3(b) shows the colour sensory response of three 
female and one male patients implanted two years ago. Their perception changed from yellow (less than 60 
Hz) to white (more than 60 Hz) when stimulation frequency varied between a few Hz and 150 Hz. Color 
perception is high by using more electrodes to create true colour bands, even if they are between two colours. 
Figure 3(c) shows colour perception from 8 electrodes in distinct retinal areas with frequency fluctuations 
(PW=0.5). Purple and yellow are colour sensations. It is noticed from blue-yellow and red-green axes 
independently under frequency changes Figure 3(d). It doesn't change much between 0 and 1.2 pulse width. 
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Figure 3. Illiterate the influence of stimulus frequency and pulse width (PW) on color perception. For 
experimental validation follows different combination of the stimulated frequency (50, 100, and 150) and 
pulse width (0.5, 1, 1.5, and 2 ms). (a) results examine that the subjected patients (2 female and | male) who 
got recent implantation, they have perception changed from yellow (less than 60 Hz) to blue (greater than 60 
Hz), when stimulation frequency changes in the range between a few Hz and 150 Hz. If more than one color 
has detected in one phosphene, they are shown in concentric rings, each representing the hue and saturation 
of one color; (b) results examine that the subjected patients (3 female and 1 males) implanted two years ago 
for them perception changed from yellow (less than 60 Hz) to white (greater than 60 Hz), when stimulation 
frequency changes in the range between a few Hz and 150 Hz; (c) color perception from 8 electrodes in 
various retinal regions of both type of implantation (recent and two years old) with frequency variations 
(PW=0.5); (d) aggregated blue and yellow scores under frequency variations, and (e) Aggregated blue and 
yellow scores under pulse width varying between 0 and 1.2 respectively 


3.3. Electrically adjusted and maintain Long-term stability towards color sensation 

Figure 4 indicates high-frequency purple enhances colour perception. Color as a dependable 
dimension of visual cues in prosthetic vision depends on the long-term stability of the electrically evoked color 
experience. As a result, compared the color perception of two type of patient based on period of implantation 
such as (i) patient of (5 months < implanted year < 10 months) and (19 months < implanted year < 30 months) 
who were subjected for analysis under different frequencies. Pooling blue and purple data. Recent implants 
may cause color-sensation issues, particularly at low and high frequencies. Argus-database II indicates that 
colour perception is independent of patient health and age, therefore automatic stimulus adjustment preserves 
medium brightness. Pre-trained CNNs employ correlation-based feature extraction to differentiate colour index 
profiles like royal yellow, lemon yellow, royal purple, lavender purple. Figure 4(a) shows the implanted 
patient's frequency-dependent colour perception. Similar yellow colour index colours become 
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indistinguishable after a few months. Electrodes don't obtain a steady current, therefore electrical stimulation 
is distributed, impairing the implanted patient's brightness. Figure 4(b) shows the implanted patient's 
frequency-dependent colour perception. People can't tell purple from blue after a few years. These two 
problems may be overcome by restoring current amplitude using updated CNN feature sets. In Figure 4(c), 
pre-trained CNNs enhance colour perception by minimising low-to-high frequency yellow-to-blue transitions 
(recent implanted patient). Low frequency yellow colour shades are artificial, however implanted patients can't 
detect true colour (yellow index) due to lack of brightness. Pre-trained CNNs recognise electrodes that need 
current amplitude adjustment and operate them with a constant quantity of current to ensure medium brightness 
for categorising related colour index issues. Figure 4(d) shows the maximum number of electrodes that can 
correctly modify electrical stimuli, yellow for low frequency. Proper electrode selection (target of 5 electrodes) 
is needed to modify electrical stimuli based on CNN feature sets. Figure 4(e) and Figure 4(f) illustrates the 


number of yellow-to-blue and blue-to-purple electrodes. 
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Figure 4. Response of color sensation (yellow to blue and blue to purple) tracked from recent and two-year- 
old implanted patient under frequency-shifting over time. (a) color perception response of the recent 
implanted patient (5 months < implanted year < 10 months) under different frequencies; (b) color perception 
response of the old implanted patient (19 months < implanted year < 30 months) under different frequencies; 
(c) enhanced color perception using pre-trained CNNs which minimize color sensation problem of yellow to 
blue transition phase at varying frequencies from low to high (recent implanted patient); (d) enhanced color 
perception using pre-trained CNNs which minimize color sensation problem of blue to purple transition 
phase at varying frequencies from low to high (old implanted patient). Electrodes (target no of electrodes 5) 
identified for adjusting electrical stimuli based on feature sets updated by pre-trained CNNs and (e) for 
yellow to blue, and (f) for blue to purple sensation respectively 


3.4. Correlation based feature extraction algorithm 


After image segmentation, feature extraction is crucial. It captures quiet characteristics from each unit 
for subsequent analysis to uncover segmented process irregularities. It uses grey level intensity to quantify 
Statistical parameters. It helps identify colour index and pattern observation problems. Figure 5 exhibits 
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correlation-based feature extraction in pre-trained CNNs. Second order statistical parameters are used to 
generate the textural feature set based on the grey level intensity (minimum 14) of each pixel and its 
neighbouring pixels. Two-dimensional matrix is used to analyse image spatial property by adjusting pixel 
intensity based on occupied frequencies. First, a pixel to the right of the reference pixel may vary its intensity 
between minimum and maximum threshold limits (i.e. 2 to 256). It also gives favourable and unfavourable 
criteria to assist attain minimal membership function quicker and pass the first phase of optimization. Because 
it involves entropy and contrast, it shows high and low pixel intensities. Here's how the training and test phase 
works. 
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Figure 5. Operational flow of correlation-based feature extraction approach involved in the pre-trained CNNs 


a. Training phase: 

- Initializing the number of iterations is carried out for finding weight vector and euclidean distance for 
effective feature selection. Approximately, set the cluster group size is equal to 30 or it will be adjusted 
to convergence range. 

- Clustering feature sets are extracted in the training phase for identifying extract feature mapping with 
ground truth images. By using equation of Igjectroqeand yas mentioned in the pseudo code are updating 
regularly based on the instant weight vector of each frame. 

- Similarly, it has continued for all other data sets in order to estimate the cost function as closer to 
desired target based on the favourable and unfavourable criterion derived from pre-trained CNN 
extraction model. 

- Once the cost function is converged, then it is fixed for selecting best threshold value by setting local 
and global factor using correlated based feature selection algorithms. 

b. Testing phase: 

- Testing phase is activated if the pre-trained CNN parameters are not converge then further updating 
is carried in the weight vector for withstanding best threshold value. Every sample output is recorded, 
and it is being validated after entire sample undergoes double optimization process. Figure 6 and 
Figure 7 show the normalised scores of the chosen electrodes (5) for higher responsiveness under blue 
to purple feeling (recently implanted patient) (old-implanted patient). Pre-trained CNN architecture 
gives enough brightness to classify colour tones. Figure 6 and Figure 7 show TPR and FPR 
improvement and maintained trade-off ratio. 
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Figure 6. Normalized scores of the targeted electrodes (5 in numbers) for better response under yellow to 
blue sensation state recorded (recent implanted patient) 
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Figure 7. Normalized scores of the targeted electrodes (5 in numbers) for better response under blue to purple 
sensation state recorded (old-implanted patient) 


4. CONCLUSION 

In this paper, an automatic recognition of color sensation with controlled phosphene brightness using 
pre-trained CNNs framework is proposed. The training features are derived from patient recorded which is 
applied to CNNs for obtaining trained datasets for better color sensation with controlled phosphene brightness 
is achieved. That implies, amount of current passing to each electrode is automatically adjusted based on the 
trained dataset receive from pre-trained CNNs framework. The feature extraction of current level thresholding 
is obtained based on patient records observed during the Argus-II test phase. Thereby, pre-trained CNNs 
provides updated feature set in order to detect the accurate electrodes who need current amplitude adjustment 
is effectively identified and operate those electrodes with standard amount of current which ensure medium 
brightness for classifying similar color index problem. The experimental results shows that the proposed system 
is achieved reasonable improvement in the transition color sensation as well as controlled brightness when 
compared with other existing systems. 
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